perf(core): optimize async operation & progress events by ido-pluto · Pull Request #23 · ido-pluto/ipull

ido-pluto · 2026-03-15T20:22:23Z

Description of change

Refactor to some core functionality of ipull that improves download speed and make ipull more light wight.

Replacing pretty-bytes & pretty-ms with an optimize version, that lower the cpu load by ~35% (!!!)
Optimizing disk writing with queue & debounce to lower the event loop load & context switch ~60% IPull 4.0.3 is heavy on the event loop #19
Debouncing event at the network file-engine level ~10%
Cacheing engine CLI calculation ~5%
--save to directory is hang and saves to current directory #21
Other bug fixes

All the performance benchmarks can be found here

https://ido-pluto.github.io/ipull-speed-test/

https://github.com/ido-pluto/ipull-speed-test

Breaking chage:

'-st' shortcut is not supported yet in the cli, use '--style' instead.
Some types has been change has I bump deps
Auto file name functionality has been change to better capture the file name

Pull-Request Checklist

Code is up-to-date with the main branch
npm run format to apply eslint formatting
npm run test passes with this change
This pull request links relevant issues as Fixes #0000
There are new or updated unit tests validating the change
Documentation has been updated to reflect this change
The new commits follow conventions explained
in CONTRIBUTING.md

- Global CLI, you do not need to wrap active downloads with downloadSequence - Auto increase parallel stream, maximize download speed - Not reusing redirected URL by default - prevent token expires for long downloads - Performance & stability improvements - Remote CLI progress, show download progress in another process easily BREAKING CHANGE: - `partsURL` removed in favor of `partURLs` - Not reusing redirected URL by default - Different chunk size based on programType - You can recall `.download()` and it will not throw an error - will return the same promise of downloading (do *not* redownload the file) - You can change the parallel stream count after download has started

# Conflicts: # package-lock.json # package.json # src/download/download-engine/download-file/download-engine-file.ts # src/download/download-engine/download-file/progress-status-file.ts # src/download/download-engine/engine/download-engine-multi-download.ts # src/download/download-engine/engine/download-engine-nodejs.ts # src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts # src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts # src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts # src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts # src/download/transfer-visualize/progress-statistics-builder.ts # src/download/transfer-visualize/transfer-cli/GlobalCLI.ts # src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts # src/download/transfer-visualize/transfer-cli/transfer-cli.ts # test/browser.test.ts # test/download.test.ts # test/fetchDownloadInfo.test.ts

Copilot

Pull request overview

This PR focuses on performance and dependency reduction in the download progress/CLI visualization and Node.js write path, while updating CI/build config and tests to match the new behavior.

Changes:

Replaced pretty-bytes / pretty-ms with new in-repo “fast” formatters and updated CLI/progress rendering to use them.
Reworked parts of the download engine runtime behavior (write buffering via WriteQueue, progress throttling, metadata-save hook changes, CLI update debouncing/caching).
Updated test setup (timeouts, fixtures) and CI build pipeline (new tsconfig.prod.json, build:prod, Vitest upgrade).

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`vitest.config.ts`	Switches to a global test timeout instead of forcing single-thread workers.
`tsconfig.prod.json`	Adds a production TypeScript build config (used by CI).
`test/utils/files.ts`	Updates large-file test fixture URL.
`test/utils/download.ts`	Adjusts local test fixture path/metadata fields.
`test/fetchDownloadInfo.test.ts`	Updates header string and snapshot formatting; removes per-suite timeout override.
`test/download.test.ts`	Tweaks chunk size and removes per-suite timeout override.
`test/copy-file.test.ts`	Removes per-suite timeout override.
`test/browser.test.ts`	Updates expected hashes; changes concurrency options; skips one browser-memory test.
`src/download/transfer-visualize/utils/prettyMSFast.ts`	Adds fast compact duration formatting.
`src/download/transfer-visualize/utils/prettyBytesFast.ts`	Adds fast byte formatting with optional locale/format controls.
`src/download/transfer-visualize/transfer-statistics.ts`	Avoids divide-by-zero when total bytes is 0.
`src/download/transfer-visualize/transfer-cli/transfer-cli.ts`	Replaces lodash debounce with manual time-based throttling + SIGINT behavior changes.
`src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts`	Switches duration formatting to the new formatter.
`src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts`	Switches byte formatting to the new formatter.
`src/download/transfer-visualize/transfer-cli/GlobalCLI.ts`	Adds caching/invalidation for displayed engines; adapts to new `updateStatues()` API.
`src/download/transfer-visualize/progress-statistics-builder.ts`	Adds common transfer-action aggregation + cached loading status.
`src/download/transfer-visualize/format-transfer-status.ts`	Switches formatting helpers to new fast formatters.
`src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts`	Introduces buffered coalescing queue for positional writes.
`src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts`	Integrates `WriteQueue`, changes metadata flush behavior, adds finalizer-based fd cleanup.
`src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts`	Cleans abort listener management; switches timeout formatting; adjusts length detection.
`src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts`	Cleans abort listener lifecycle; switches timeout formatting; adjusts length detection.
`src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts`	Adds cloned-listener cleanup to prevent leaks.
`src/download/download-engine/engine/download-engine-nodejs.ts`	Adapts save-progress hook to new write-stream metadata API.
`src/download/download-engine/engine/download-engine-multi-download.ts`	Ensures `close()` happens on error path too.
`src/download/download-engine/download-file/progress-status-file.ts`	Replaces `ProgressStatusFile` class usage with `EMPTY_PROGRESS_STATUS` constant.
`src/download/download-engine/download-file/download-engine-file.ts`	Reworks progress status creation, adds progress throttling, changes save-progress hook shape.
`src/cli/cli.ts`	Adjusts save-path interpretation and output naming behavior.
`README.md`	Improves browser usage docs around CORS/range + manual download info.
`package.json`	Adds `build:prod`, upgrades dependencies (Vitest, xmlhttprequest-ssl, semantic-release), removes pretty-* deps.
`.github/workflows/test.yml`	Uses `build:prod` in CI.
`.github/workflows/build.yml`	Uses `build:prod` in CI; updates release job environment usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/cli/cli.ts

...d/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts

...oad/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts

src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts

.../download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts

src/download/download-engine/download-file/download-engine-file.ts

src/download/transfer-visualize/utils/prettyBytesFast.ts

Copilot

Pull request overview

This PR modernizes the project’s build/test tooling and refactors download progress/reporting internals for performance and reliability, including replacing third-party “pretty” formatters with internal fast implementations and introducing a new buffered write queue for Node.js writes.

Changes:

Upgrade tooling (Vitest v4, new tsconfig.prod.json, CI uses build:prod, update test timeouts).
Replace pretty-bytes / pretty-ms with new internal prettyBytesFast / prettyMSFast and update CLI progress formatting.
Refactor Node.js write stream to use a new WriteQueue and adjust progress saving / CLI update behavior.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
vitest.config.ts	Removes worker pinning and sets a global test timeout.
tsconfig.prod.json	Adds a production TS build config used by CI.
test/utils/files.ts	Changes the “big file” test fixture URL.
test/utils/download.ts	Updates big-file filename and augments test download info payload.
test/fetchDownloadInfo.test.ts	Adjusts headers formatting and relies on global timeouts.
test/download.test.ts	Updates chunk size and relies on global timeouts.
test/copy-file.test.ts	Relies on global timeouts.
test/browser.test.ts	Updates snapshots, concurrency options, and skips XHR memory test.
src/download/transfer-visualize/utils/prettyMSFast.ts	Adds a compact millisecond formatter.
src/download/transfer-visualize/utils/prettyBytesFast.ts	Adds a fast pretty-bytes replacement (+ trunc formatter).
src/download/transfer-visualize/transfer-statistics.ts	Fixes divide-by-zero percentage when total is 0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts	Replaces lodash debounce with a manual debounce + progress getter.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts	Switches to internal pretty-ms formatter.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts	Switches to internal pretty-bytes formatter.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts	Adds CLI engine/status caching and integrates new TransferCli API.
src/download/transfer-visualize/progress-statistics-builder.ts	Improves aggregated status and uses shared empty progress status.
src/download/transfer-visualize/format-transfer-status.ts	Switches to internal formatters; changes percentage formatting logic.
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts	Introduces a coalescing buffered writer with parallel positional writes.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts	Refactors Node write stream to use `WriteQueue` + deferred metadata flush.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts	Improves timeout message formatting and content-length handling.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts	Cleans abort listener lifecycle; improves content-length logic; updates timeout msg formatting.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts	Adds cleanup for cloned-state listeners to prevent leaks.
src/download/download-engine/engine/download-engine-nodejs.ts	Updates progress-save hook to the new sync `onSaveProgress` flow.
src/download/download-engine/engine/download-engine-multi-download.ts	Ensures close occurs on error and tweaks finish sequence.
src/download/download-engine/download-file/progress-status-file.ts	Replaces `ProgressStatusFile` class with `EMPTY_PROGRESS_STATUS`.
src/download/download-engine/download-file/download-engine-file.ts	Refactors status construction, throttles progress events, changes save callback API.
src/cli/cli.ts	Reworks `--save` path handling and file naming behavior.
README.md	Clarifies CORS/range-request workaround usage and documents manual download info.
package.json	Adds `build:prod`, upgrades vitest/xmlhttprequest-ssl/semantic-release, removes pretty-* deps.
.github/workflows/test.yml	Runs `build:prod` in CI instead of `build`.
.github/workflows/build.yml	Runs `build:prod` and adjusts release env vars.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/download/transfer-visualize/utils/prettyBytesFast.ts

src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts

.../download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts

src/cli/cli.ts

.github/workflows/build.yml

Copilot

Pull request overview

This PR focuses on improving download/CLI performance and reliability by replacing some third-party formatting utilities with faster local implementations, introducing a new buffered write queue for Node.js writes, and updating CI/test configuration to match the new build/test behavior.

Changes:

Added high-performance prettyBytes/prettyMS utilities and wired them into transfer/CLI rendering.
Introduced a new WriteQueue and refactored the Node.js write stream + progress saving behavior.
Updated CI to use a new build:prod (via tsconfig.prod.json) and refreshed tests/configs for timeouts and fixtures.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
vitest.config.ts	Removes single-worker constraints; sets a global test timeout.
tsconfig.prod.json	Adds a production TS build config (no sourcemaps, emit to dist).
test/utils/files.ts	Switches BIG_FILE fixture URL to a different large remote file.
test/utils/download.ts	Updates big-file example extension and download metadata fields.
test/fetchDownloadInfo.test.ts	Adjusts headers/snapshots and removes suite timeout override.
test/download.test.ts	Tweaks chunk size and removes suite timeout override.
test/copy-file.test.ts	Removes suite timeout override.
test/browser.test.ts	Updates snapshots, changes concurrency/repeats usage, and skips an xhr memory test.
src/download/transfer-visualize/utils/prettyMSFast.ts	Adds fast compact milliseconds formatter.
src/download/transfer-visualize/utils/prettyBytesFast.ts	Adds fast pretty-bytes implementation (number+bigint + options).
src/download/transfer-visualize/transfer-statistics.ts	Fixes percentage calculation when total is zero.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts	Reworks CLI update throttling/debouncing and SIGINT behavior.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts	Switches to internal prettyMS formatter.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts	Switches to internal prettyBytes formatter.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts	Adds caching + update getter approach for CLI status collection.
src/download/transfer-visualize/progress-statistics-builder.ts	Improves multi-engine aggregation and caches loading status formatting.
src/download/transfer-visualize/format-transfer-status.ts	Switches formatting to internal prettyBytes/prettyMS + trunc formatting.
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts	Introduces a new coalescing/parallel positional write buffer.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts	Refactors Node.js write stream to use WriteQueue + deferred metadata flush.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts	Cleans up abort handlers and improves length detection with encoding/range logic.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts	Ensures abort listener cleanup and improves length detection with encoding/range logic.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts	Adds cloned-state listener cleanup to avoid leaks across retries/clones.
src/download/download-engine/engine/download-engine-nodejs.ts	Updates progress-saving hook to the new write-stream metadata flush API.
src/download/download-engine/engine/download-engine-multi-download.ts	Ensures close is called on failure and tweaks completion flow.
src/download/download-engine/download-file/progress-status-file.ts	Replaces ProgressStatusFile class with a shared EMPTY_PROGRESS_STATUS constant.
src/download/download-engine/download-file/download-engine-file.ts	Refactors status building, throttles progress events, and changes save callback shape.
src/cli/cli.ts	Improves `--save` path handling for directory vs file and multi-file naming.
README.md	Clarifies CORS/range options and documents manual download info override.
package.json	Adds `build:prod`, updates tool versions, removes pretty-* deps, upgrades vitest/xmlhttprequest-ssl.
.github/workflows/test.yml	Uses `build:prod` in CI typecheck step.
.github/workflows/build.yml	Uses `build:prod` and updates release env setup.

Comments suppressed due to low confidence (1)

src/download/transfer-visualize/transfer-cli/transfer-cli.ts:5

lodash.debounce was removed from this file, and a repo-wide search shows no remaining usages. Consider removing the lodash.debounce dependency from package.json as well to avoid shipping/maintaining an unused dependency.

import UpdateManager from "stdout-update";
import {TransferCliProgressBar} from "./progress-bars/base-transfer-cli-progress-bar.js";
import cliSpinners from "cli-spinners";
import {FormattedStatus} from "../format-transfer-status.js";
import switchCliProgressStyle from "./progress-bars/switch-cli-progress-style.js";

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/build.yml

src/download/download-engine/download-file/download-engine-file.ts

src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts

.../download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts

test/fetchDownloadInfo.test.ts

BREAKING CHANGE: shortcut '-st' flag is no support anymore use '--style' instead

BREAKING CHANGE: 'size' changed to remoteSize and downloadSize

Copilot

Pull request overview

This PR refactors ipull’s core download and CLI progress pipeline to reduce event-loop load and CPU usage while adding more deterministic tests and a production-focused TS build.

Changes:

Replace pretty-bytes/pretty-ms with faster internal formatters and reduce CLI progress update overhead (debounce/throttle + engine/status caching).
Optimize Node.js disk writes via a buffered positional WriteQueue, and adjust progress/save event emission to be less chatty.
Expand/refresh Vitest coverage with a local HTTP test server and new edge-case tests; add tsconfig.prod.json and switch CI builds to build:prod.

Reviewed changes

Copilot reviewed 57 out of 59 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
vitest.config.ts	Adjust Vitest runtime settings (timeout; worker config removed).
tsconfig.prod.json	Add prod build TS config emitting declarations to `dist/`.
test/utils/local-server.ts	New local Express server for deterministic download tests.
test/utils/files.ts	Update “big file” test URL.
test/utils/fetchRangeDownloadParts.ts	Utility to fetch multiple byte ranges in parallel.
test/utils/download.ts	Test helpers updated for new `DownloadFilePart` shape and options.
test/redirect-handling.test.ts	New redirect handling tests.
test/range-validation-error.test.ts	New range validation error tests.
test/range-download-parts.test.ts	New test for multi-part range download assembly.
test/progress-events.test.ts	New progress/event emission test.
test/pause-resume-abort.test.ts	New pause/resume/abort behavior tests.
test/parallel-streams-edge.test.ts	New parallel stream edge/stress tests.
test/mixed-parts.test.ts	New mixed range/non-range parts test.
test/fetchstream-error-propagation.test.ts	New test ensuring fetch-stream errors propagate without hangs.
test/fetchDownloadInfo.test.ts	Refactor to use local server instead of external throttled proxy.
test/download.test.ts	Update tests to use new file-part fields and options.
test/daynamic-content-length.test.ts	Update assertions to new Vitest API style.
test/copy-file.test.ts	Update assertions to new Vitest API style.
test/content-length-mismatch.test.ts	New tests for content-length mismatch handling.
test/browser.test.ts	Update expected hashes and tweak concurrency options.
test/accept-range-false.test.ts	New tests for accept-range=false and content-length=0 cases.
src/download/transfer-visualize/utils/prettyMSFast.ts	New fast compact milliseconds formatter.
src/download/transfer-visualize/utils/prettyBytesFast.ts	New fast bytes formatter (plus trunc formatting helper).
src/download/transfer-visualize/transfer-statistics.ts	Fix percentage calculation when `total===0`.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts	Replace lodash debounce with time-based throttling and safer SIGINT behavior.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts	Switch to internal pretty-ms replacement.
src/download/transfer-visualize/transfer-cli/progress-bars/base-transfer-cli-progress-bar.ts	Type import adjustments for cli-spinners.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts	Switch to internal pretty-bytes replacement.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts	Cache CLI engines list and adjust lifecycle around multi-download completion.
src/download/transfer-visualize/progress-statistics-builder.ts	Reduce allocations; cache loading status; track common transfer action.
src/download/transfer-visualize/format-transfer-status.ts	Use internal formatters and trunc percent formatting.
src/download/download-engine/types.ts	Change part shape (`remoteFileSize`, `downloadSize`, `range`, `fetchStream`, per-part options).
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts	New buffered write queue for coalescing/parallel positional writes.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts	Replace debounce writer with `WriteQueue`; metadata flush changes; FD lifecycle updates.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-browser.ts	Blob creation typing adjustment.
src/download/download-engine/streams/download-engine-fetch-stream/utils/smart-chunk-split.ts	Rename field usage to `downloadSize`.
src/download/download-engine/streams/download-engine-fetch-stream/utils/content-disposition.ts	Trim trailing `;` before parsing content-disposition.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts	Add header-timeout abort, stricter length checks, and updated error formatting.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-local-file.ts	Use a dedicated lock scope for file open.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts	Add header-timeout abort, improved length checks, and dynamic start/end behavior.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts	Add range support and header timeout abort controller; listener cleanup.
src/download/download-engine/engine/error/RangeOutOfPartLengthError.ts	New engine error for invalid part ranges.
src/download/download-engine/engine/download-engine-nodejs.ts	Rework part URL input to internal “full part URL” objects; metadata save hook update; save-path fixes.
src/download/download-engine/engine/download-engine-multi-download.ts	Ensure `close()` is called on error paths.
src/download/download-engine/engine/download-engine-browser.ts	Rework part URL input to internal “full part URL” objects.
src/download/download-engine/engine/base-download-engine.ts	Add range-aware part creation and internal part-url normalization.
src/download/download-engine/download-file/progress-status-file.ts	Replace class with an `EMPTY_PROGRESS_STATUS` object baseline.
src/download/download-engine/download-file/download-programs/switch-program.ts	Pass `parallelStreams` explicitly into programs.
src/download/download-engine/download-file/download-programs/download-program-stream.ts	Optimize `parallelStreams===1` behavior.
src/download/download-engine/download-file/download-programs/download-program-chunks.ts	Update constructor signature to accept `parallelStreams`.
src/download/download-engine/download-file/download-programs/base-download-program.ts	Track `parallelStreams` independently of saved progress.
src/download/download-engine/download-file/download-engine-file.ts	Per-part options, progress throttling, and revised save/progress flow.
src/cli/cli.ts	Fix `--save` directory/file handling; remove `-st` shortcut for style.
README.md	Clarify CORS/range workaround docs; show `defaultFetchDownloadInfo` usage.
package.json	Add `build:prod`; bump deps (Vitest/TS/etc.); remove pretty-* deps.
.gitignore	Ignore new large test artifacts.
.github/workflows/test.yml	Switch TS step to `npm run build:prod`.
.github/workflows/build.yml	Switch build to `npm run build:prod`; remove `NPM_TOKEN` from release env.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...ad/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts

    protected get _endSize() {
-        return Math.min(this.state.endChunk * this.state.chunkSize, this.state.activePart.size);
+        const rangeEnd = this.options.range!.end >= 0 ? this.options.range!.end + 1 : Infinity;
+        return Math.min(this._startSize + this.state.endChunk * this.state.chunkSize, rangeEnd, this.state.activePart.remoteFileSize);


src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts

+    private _tryMerge(cursor: number, buffers: Uint8Array[], length: number): boolean {
+        for (let i = 0; i < this._regions.length; i++) {
+            const region = this._regions[i];
+            const regionEnd = region.cursor + region.length;
+
+            if (cursor === regionEnd) {
+                region.buffers.push(...buffers);
+                region.length += length;
+            } else if (cursor + length === region.cursor) {
+                region.cursor = cursor;
+                region.buffers.unshift(...buffers);
+                region.length += length;
+            } else {
+                continue;
+            }
+
+            if (this._tryMerge(region.cursor, region.buffers, region.length)) {
+                this._regions.splice(i, 1);
+            }
+        }
+
+        return false;
+    }


test/utils/local-server.ts

+    app.get("/bad-length.bin", (req, res) => {
+        const rangeHeader = req.header("range");
+        if (!rangeHeader) {
+            setCommonHeaders(res, true, TEST_FILE_SIZE + 1000); // wrong length deliberately
+            res.status(200).send(TEST_FILE_DATA.slice(0, TEST_FILE_SIZE + 1000)); // but send correct data? Wait, to mismatch, send less or more.
+            // To mismatch, perhaps send TEST_FILE_DATA, but header wrong.
+            // But to make it throw, send TEST_FILE_DATA, header wrong, but in fetch, header wrong, throw.
+            // In xhr, data length correct, header wrong, but xhr checks data vs expected, expected from header? No, expected is calculated, data is correct, so no throw.
+            // For no range, the expected is the totalSize, contentLength from header.
+            // So for no range, if header wrong, in fetch, contentLength !== expected, throw.
+            // In xhr, the check is if expected !== arrayBuffer.byteLength, but expected is from header? No, in xhr, expectedContentLength is calculated from range, same as fetch.
+            // In xhr, the check is expectedContentLength !== arrayBuffer.byteLength
+            // So for no range, expected = total, data length = total, header wrong, but check is data vs expected, not header.
+            // So for no range, it won't throw in xhr.
+            // For range, in xhr, data length = sent length = chunk.length - 1, expected = chunk.length, throw.
+            // For fetch, for range, header = chunk.length - 1, expected = chunk.length, throw.
+            // For no range, in fetch, header = TEST_FILE_SIZE + 1000, expected = TEST_FILE_SIZE, throw.
+            // In xhr, for no range, data length = TEST_FILE_SIZE, expected = TEST_FILE_SIZE, no throw.
+            // So to make it throw in both, for no range, send wrong data length.
+            // So set header to TEST_FILE_SIZE + 1000, send TEST_FILE_DATA (length TEST_FILE_SIZE).
+            // In fetch, header wrong, throw.
+            // In xhr, data length = TEST_FILE_SIZE, expected = TEST_FILE_SIZE + 1000, throw.
+            // Yes.
+            res.status(200).send(TEST_FILE_DATA);
+            return;
+        }


test/utils/local-server.ts

+            res.status(200).send(TEST_FILE_DATA.slice(0, TEST_FILE_SIZE + 1000)); // but send correct data? Wait, to mismatch, send less or more.
+            // To mismatch, perhaps send TEST_FILE_DATA, but header wrong.
+            // But to make it throw, send TEST_FILE_DATA, header wrong, but in fetch, header wrong, throw.
+            // In xhr, data length correct, header wrong, but xhr checks data vs expected, expected from header? No, expected is calculated, data is correct, so no throw.
+            // For no range, the expected is the totalSize, contentLength from header.
+            // So for no range, if header wrong, in fetch, contentLength !== expected, throw.
+            // In xhr, the check is if expected !== arrayBuffer.byteLength, but expected is from header? No, in xhr, expectedContentLength is calculated from range, same as fetch.
+            // In xhr, the check is expectedContentLength !== arrayBuffer.byteLength
+            // So for no range, expected = total, data length = total, header wrong, but check is data vs expected, not header.
+            // So for no range, it won't throw in xhr.
+            // For range, in xhr, data length = sent length = chunk.length - 1, expected = chunk.length, throw.
+            // For fetch, for range, header = chunk.length - 1, expected = chunk.length, throw.
+            // For no range, in fetch, header = TEST_FILE_SIZE + 1000, expected = TEST_FILE_SIZE, throw.
+            // In xhr, for no range, data length = TEST_FILE_SIZE, expected = TEST_FILE_SIZE, no throw.
+            // So to make it throw in both, for no range, send wrong data length.
+            // So set header to TEST_FILE_SIZE + 1000, send TEST_FILE_DATA (length TEST_FILE_SIZE).
+            // In fetch, header wrong, throw.
+            // In xhr, data length = TEST_FILE_SIZE, expected = TEST_FILE_SIZE + 1000, throw.
+            // Yes.


vitest.config.ts

 export default defineConfig({
    test: {
        pool: "threads",
-        maxWorkers: 1,
-        minWorkers: 1,
-        poolOptions: {
-            threads: {
-                minThreads: 1,
-                maxThreads: 1
-            }
-        }
+        testTimeout: 1000 * 60 * 3
    }


test/redirect-handling.test.ts

+// This URL should redirect. Replace with a known redirecting URL if needed.
+const REDIRECT_URL = "https://httpbin.org/redirect-to?url=https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_92x30dp.png";
+


test/progress-events.test.ts

+const TEST_URL = "https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_92x30dp.png";
+


ido-pluto and others added 25 commits May 22, 2025 23:51

fix(multiDownload): update status when skipping download

7296216

fix(multiDownload): events & cli progress

5ad092d

pref(microtask): optimize timers

2504676

fix(transferStatistics): 0 when download size is unknown

9826396

pref(downloadInfo): optimize requests

fc48e3b

docs(README): typo

a6b98f5

fix(build): no source map in prod

6954781

feat(downloadURL): regenerate download url when token expires

dee3ad5

fix(cli): save flag does not save to directory

872c309

fix: tests

6228d28

Merge branch 'main' into feat/speed-ipull

c3347cc

fix: release

bcd1870

fix(write-stream): finalization close file handler

d8cd69a

feat(GlobalCLI): cache engine order generation

a4ca68c

feat(write-stream): parallel write queue

64de83d

feat(cli): faster event creation

4ff195e

fix(download-stream): event cleanup

264f918

fix(progress): dedup engine on progress building

6b1a994

fix(write-queue): close fd

9fc3cc8

fix: format

696c8c7

fix: remove unused var

2b648e7

fix: remove unused file

18ef713

fix(multi-engine): close before throw

31c4a7e

ido-pluto requested a review from Copilot March 15, 2026 20:51

Copilot started reviewing on behalf of ido-pluto March 15, 2026 20:52 View session

Copilot AI reviewed Mar 15, 2026

View reviewed changes

ido-pluto added 2 commits March 15, 2026 22:05

fix(cli): file indexing

cd641ad

fix(download-size): get range request to fetch size

1143ae5

ido-pluto added 2 commits March 15, 2026 22:24

fix(write-queue): default buffer size

a94bd9b

fix(write-stream): catch close errors

26e3a4d

ido-pluto requested a review from Copilot March 15, 2026 21:30

Copilot started reviewing on behalf of ido-pluto March 15, 2026 21:31 View session

Copilot AI reviewed Mar 15, 2026

View reviewed changes

ido-pluto added 6 commits March 16, 2026 10:07

fix(write-queue): prevent flashing the same region twice

df7a24d

fix(pretty-bytes-fast): large numbers

f009e31

fix(cli): check if path is directory by fs

011a8bc

fix(pretty-bytes-fast): bigint division

b265372

fix(pretty-bytes-fast): export formatTrunc

81bb92e

format

b9faef4

ido-pluto requested a review from Copilot March 16, 2026 09:46

Copilot started reviewing on behalf of ido-pluto March 16, 2026 09:46 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

ido-pluto added 7 commits March 16, 2026 11:20

fix(name): content disposition parsing

4c1ae53

bump: deps

57a0762

BREAKING CHANGE: shortcut '-st' flag is no support anymore use '--style' instead

fix(write-queue): better merging

1c95ee1

feat: parts range download

fab19ee

BREAKING CHANGE: 'size' changed to remoteSize and downloadSize

fix(chores): typedoc

d3c3f34

fix(bump): docs generation

b5d25a8

fix: format

9a0b455

ido-pluto requested a review from Copilot April 11, 2026 07:52

Copilot started reviewing on behalf of ido-pluto April 11, 2026 07:52 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

		// This URL should redirect. Replace with a known redirecting URL if needed.
		const REDIRECT_URL = "https://httpbin.org/redirect-to?url=https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_92x30dp.png";

		const TEST_URL = "https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_92x30dp.png";

Uh oh!

Conversation

ido-pluto commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of change

All the performance benchmarks can be found here

Pull-Request Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ido-pluto commented Mar 15, 2026 •

edited

Loading